1 P-Values

Suppose ${\displaystyle \varphi (X)}$ rejects for large values of ${\displaystyle T(X)}$. We can informally define p-value as the "under null hypothesis, probability that ${\displaystyle T(X)}$ is as large or larger than what we observed". I.e. $$p(x)={\mathbb{P}}_{H_0}(T(X)\ge T(x))=\underset{\theta \in {\mathrm{\Theta }}_0}{sup}{\mathbb{P}}_{\theta }(T(X)\ge T(x)).$$

Now we give a formal definition:

${\displaystyle \varphi }$ here measures how "extreme" an observed ${\displaystyle T(X)}$ is.

For ${\displaystyle \theta \in {\mathrm{\Theta }}_0}$, $$\begin{array}{rl}{\mathbb{P}}_{\theta }(p(x)\le \alpha )& ={\mathbb{P}}_{\theta }(sup\{\tilde{\alpha }|{\varphi }_{\tilde{\alpha }}(x)<1\}\le \alpha )\\ & \le \underset{\epsilon \to 0^{+}}{lim}{\mathbb{P}}_{\theta }({\varphi }_{\alpha +\epsilon }(x)=1)\\ & \le \underset{\epsilon \to 0^{+}}{lim}(\alpha +\epsilon )=\alpha .\end{array}$$
So p-value stochastically dominates ${\displaystyle u[0,1]}$.
If ${\displaystyle {\varphi }_{\alpha }}$ rejects for large ${\displaystyle T(X)}$, reduces to original definition.

Note the p-value is defined based on

• The model ${\displaystyle P_{\theta }}$ and hypothesis ${\displaystyle H_0}$;
• The data ${\displaystyle X}$
• The choice of test.

2 Confidence Sets

2.1 Definition

• ${\displaystyle C(X)}$ is random, not ${\displaystyle g(\theta )}$. ${\displaystyle g(\theta )}$ is a deterministic function of ${\displaystyle \theta }$.
• Often misinterpreted as Bayesian guarantee.
• say "${\displaystyle C(X)}$ has a ${\displaystyle 95\mathrm{\%}}$ chance of covering", not "${\displaystyle g(\theta )}$"

2.2 Duality of Testing & Confidence Sets

Suppose we have a level- ${\displaystyle \alpha }$ test ${\displaystyle \varphi (x;a)}$ of $$\begin{array}{}\text{(2.1)}& H_0:g(\theta )=a\text{ vs }{H}_1:g(\theta )\ne a,\mathrm{\forall }a\in g(\mathrm{\Theta }).\end{array}$$
We can use it to make a confidence set for ${\displaystyle g(\theta )}$:
Let ${\displaystyle C(X)=\{a|\varphi (x;a)<1\}}$ (all non-rejected values of ${\displaystyle \theta }$). Then $${\mathbb{P}}_{\theta }(C(X)\not\ni g(\theta ))={\mathbb{P}}_{\theta }(\varphi (x;g(\theta ))=1)\le \alpha ,\mathrm{\forall }\theta .$$
Alternatively, suppose ${\displaystyle C(X)}$ is a ${\displaystyle 1-\alpha }$ confidence set for ${\displaystyle g(\theta )}$. We can use ${\displaystyle C}$ to construct a test ${\displaystyle \varphi (X)}$ of (2.1): let ${\displaystyle \varphi (X)=\mathbf{1}\{a\notin C(X)\}}$. For ${\displaystyle \theta :g(\theta )=a}$, $${\mathbb{E}}_{\theta }\varphi (X)={\mathbb{P}}_{\theta }(C(X)\not\ni g(\theta ))\le \alpha .$$ This is called inverting the test.

2.3 Confidence Interval for Median

For nonparametric model ${\displaystyle X_1,\cdots ,X_n\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }F}$, (${\displaystyle F}$ is any c.d.f) Define ${\displaystyle g(F)=\mathrm{median}(F)=F^{-1}\left(\frac{1}{2}\right)}$. Consider two-sided test $$\begin{array}{rl}& H_0:g(F)=\mu ⟺F(\mu )=\frac{1}{2}\\ \text{ vs }& H_1:g(F)\ne \mu ⟺F(\mu )\ne \frac{1}{2}.\end{array}$$
Denote ${\displaystyle S(X;\mu )=\mathrm{\#}\{X_i>\mu \}\sim \mathrm{Binomial}(n,1-F(\mu ))\stackrel{H_0}{=}\frac{1}{2}}$. Reject for ${\displaystyle T(X;\mu )=|S(X;\mu )-\frac{n}{2}|>c_{\alpha }}$. Then $$\begin{array}{rl}\mu \in C(X)⟺& |S(X;\mu )-\frac{n}{2}|\le c_{\alpha }\\ ⟺& \mathrm{\#}\{X_i>\mu \}\in [\frac{n}{2}-c_{\alpha },\frac{n}{2}+c_{\alpha }]\\ ⟺& \mu \in [X_{(\frac{n}{2}-c_{\alpha })},X_{(\frac{n}{2}+c_{\alpha })}].\end{array}$$

3 Confidence Intervals/Bounds

If ${\displaystyle C(X)=[C_1(X),C_2(X)]}$, we say ${\displaystyle C(X)}$ is a confidence interval (CI).

• ${\displaystyle C(X)=[C_1(X),\mathrm{\infty })}$ is lower confidence bound (LCB).
• ${\displaystyle C(X)=(-\mathrm{\infty },C(X)]}$ is upper confidence bound (UCB).

We usually get LCB/UCB by inverting a one-sided test in appropriate direction called uniformly most accurate (UMA) if test UMP. And get CI by inverting a two-sided test called UMAU if test is UMPU.